Pose-conditioned convolutional generative models struggle with high-quality 3D-consistent image generation from single-view datasets, due to their lack of sufficient 3D priors. Recently, the integration of Neural Radiance Fields (NeRFs) and generative models, such as Generative Adversarial Networks (GANs), has transformed 3D-aware generation from single-view images. NeRF-GANs exploit the strong inductive bias of neural 3D representations and volumetric rendering at the cost of higher computational complexity. This study aims at revisiting pose-conditioned 2D GANs for efficient 3D-aware generation at inference time by distilling 3D knowledge from pretrained NeRF-GANs. We propose a simple and effective method, based on re-using the well-disentangled latent space of a pre-trained NeRF-GAN in a pose-conditioned convolutional network to directly generate 3D-consistent images corresponding to the underlying 3D representations. Experiments on several datasets demonstrate that the proposed method obtains results comparable with volumetric rendering in terms of quality and 3D consistency while benefiting from the computational advantage of convolutional networks. The code will be available at: https://github.com/mshahbazi72/NeRF-GAN-Distillation

Untrained convolutional neural networks have emerged as highly successful tools for image recovery and restoration, for a variety of problems including denoising, compressive sensing, and inpainting [Uly 18; Jin 19; Vee 18; JH19; Hec19; HH19; Bos 20; Wan 20; HA20; Aro 20]. As opposed to trained convolutional neural networks, that learn an image prior from training data, untrained convolutional networks act as an image prior without any training and solely based on the architecture of the network and the optimization procedure used to fit them. The benefit of untrained networks was first observed in the Deep Image Prior (DIP) paper [Uly 18]. The key observation of Ulyanov et al. [Uly 18] is that fitting a standard overparameterized convolutional autoencoder (specifically, the U-net [Ron 15] or variations thereoff) to a single noisy/corrupted image, when combined with early stopping, yields excellent denoising, inpainting, and super-resolution performance. Subsequent literature has demonstrated that many elements of the architecture of a convolutional autoencoder--such as the encoder part--are irrelevant for this behavior to emerge.

Denoising and Regularization via Exploiting the Structural Bias of Convolutional Generators Reinhard Heckel and Mahdi Soltanolkotabi † Dept. of Electrical and Computer Engineering, Technical University of Munich † Dept. of Electrical and Computer Engineering, University of Southern California November 1, 2019 Abstract Convolutional Neural Networks (CNNs) have emerged as highly successful tools for image generation, recovery, and restoration. This success is often attributed to large amounts of training data. However, recent experimental findings challenge this view and instead suggest that a major contributing factor to this success is that convolutional networks impose strong prior assumptions about natural images. A surprising experiment that highlights this architectural bias towards natural images is that one can remove noise and corruptions from a natural image without using any training data, by simply fitting (via gradient descent) a randomly initialized, over-parameterized convolutional generator to the single corrupted image. While this over-parameterized network can fit the corrupted image perfectly, surprisingly after a few iterations of gradient descent one obtains the uncorrupted image. This intriguing phenomena enables state-of-the-art CNN-based denoising and regularization of linear inverse problems such as compressive sensing. In this paper we take a step towards demystifying this experimental phenomena by attributing this effect to particular architectural choices of convolutional networks, namely convolutions with fixed interpolating filters. We then formally characterize the dynamics of fitting a two layer convolutional generator to a noisy signal and prove that early-stopped gradient descent denoises/regularizes. This results relies on showing that convolutional generators fit the structured part of an image significantly faster than the corrupted portion. 1 Introduction Convolutional neural networks are extremely popular for image generation. The majority of image generating networks is convolutional, ranging from Deep Convolutional Generative Adversarial Networks (DC-GANs) [Rad 15] to the U-Net [Ron 15]. It is well known that convolutional neural networks incorporate implicit assumption about the signals they generate, such as pixels that are close being related. This makes them particularly well suited for representing sets of images or modeling distributions of images. It is less known, however, that those prior assumptions build into the architecture are so strong that convolutional neural networks are useful even without ever being exposed to training data. The latter was first shown in the Deep Image Prior (DIP) paper [Uly 18]. Ulyanov et al. [Uly 18] observed that when'training' an standard convolutional auto-encoder such as the popular U-net [Ron 15] on a single noisy image and regularizing by early stopping, the network performs image restoration such as denoising with state-of-the-art performance.

Deep convolutional neural networks trained on large datsets have emerged as an intriguing alternative for compressing images and solving inverse problems such as denoising and compressive sensing. However, it has only recently been realized that even without training, convolutional networks can function as concise image models, and thus regularize inverse problems. In this paper, we provide further evidence for this finding by studying variations of convolutional neural networks that map few weight parameters to an image. The networks we consider only consist of convolutional operations, with either fixed or parameterized filters followed by ReLU non-linearities. We demonstrate that with both fixed and parameterized convolutional filters those networks enable representing images with few coefficients. What is more, the underparameterization enables regularization of inverse problems, in particular recovering an image from few observations. We show that, similar to standard compressive sensing guarantees, on the order of the number of model parameters many measurements suffice for recovering an image from compressive measurements. Finally, we demonstrate that signal recovery with a un-trained convolutional network outperforms standard l1 and total variation minimization for magnetic resonance imaging (MRI).